Reduce memtory usage #83
Conversation
Bring up to date
tweaks to reduce memory use, still WIP
| cols_to_keep = c( | ||
| "name", # not used currently but could be handy | ||
| "distances", | ||
| "gradient_smooth", |
There was a problem hiding this comment.
We ultimately need that column. Fine if it works, surprised if it does after this change though.
Robinlovelace
left a comment
Robinlovelace
left a comment
There was a problem hiding this comment.
Great to see this attempt to reduce memory usage. My thinking is that query could be used to ignore all keys we don't use. Any ideas about impact on memory use after this change in any case? Benchmark could help.
| res = readr::read_csv(file, show_col_types = FALSE) | ||
| n_char = nchar(res$json) | ||
|
|
||
| res = data.table::fread(file, select = "json") |
There was a problem hiding this comment.
I agree with this, readr, which uses vroom, seems to be unreliable.
| } | ||
| } | ||
|
|
||
| results = RcppSimdJson::fparse(results_raw, query = "/marker", query_error_ok = TRUE, always_list = TRUE) |
There was a problem hiding this comment.
Is there a way to reduce what is read-in here with a different query?
| x$SPECIALIDFORINTERNAL2 <- NULL | ||
| cols = cols_to_keep %in% names(x) | ||
| x[cols_to_keep] | ||
| cols_to_keep3 = unique(c(cols_to_keep,"gradient_segment","elevation_change","gradient_smooth")) |
There was a problem hiding this comment.
What if any of these are not needed? For NPT gradient_smooth is the only one we need.
|
|
||
| get_values = function(v, fun) { | ||
| sapply(v, function(x) fun(as.numeric(x))) | ||
| vapply(v, function(x) fun(as.numeric(x)), 1) |
There was a problem hiding this comment.
Same outcome, what's the advantage?
| } | ||
|
|
||
| extract_values = function(x) stringr::str_split(x, pattern = ",") | ||
| extract_values = function(x) stringi::stri_split_fixed(x, pattern = ",") |
There was a problem hiding this comment.
Sounds reasonable, what's the thinking behind this.
|
Can you try to make actions happy also Malcolm? |
2 participants
WIP so just a place holder, but a few tweaks to reduce memory usage